Do we need “Harmless” Bayesian Optimization and “First-Order” Bayesian Optimization?

A recent empirical study highlighted the shocking result that, for many hyperparameter tuning problems, Bayesian optimization methods can be outperformed by random guessing run for twice as many iterations [1]. This is supported by theoretical results showing the optimality of random search under certain assumptions, but disagrees with other theoretical and empirical results showing that Bayesian optimization can lead to large gains in some situations. In light of this fact, we propose two research directions that we believe the community should pursue. First, we should focus on developing “harmless” Bayesian optimization methods that do no worse than random, and we propose a very simple “harmless” algorithm. Second, we should focus on developing first-order Bayesian optimization algorithms that use gradient information to improve performance for situations where Bayesian optimization already beats random. We empirically show the advantage of both of these ideas in simple simulations. We also propose a simple strategy for reducing the memory and computational requirements of existing first-order Bayesian optimization methods by using directional derivatives instead of full gradients, which can be obtained from analytic functions even when gradient code is not available.